ELECTRONIC EQUIPMENT AND POINTER DISPLAY METHOD 



BACKGROUND OF THE INVENTION 

5 The present invention relates to a man-machine interface 

technique, and more particularly to a graphical user interface which 
makes use of a plurality of pointing devices. 

In a man-machine interface of a computer system, the GUI 
O 10 (Graphical User Interface) which makes use of a pointing device has been 
5 widely used. In the general GUI, windows, various kinds of objects and 

IS a pointer which corresponds to the pointing device and the like are 

ffi 

ijj displayed on a screen and inputs are received from the pointing device 

L. manipulated by a user. 

0 15 Recently, along with the progress of the improvement of the user 

CP interface in the computer system, various ideas have been proposed to 

enable a manipulation with a feeling close to the real world even in the 
GUI. For example, at the time of deleting unnecessary files, drugging 
and dropping of these files into an icon indicating a litter bin is performed 
20 as if litters are actually thrown into a litter bin in a real space. 

SUMMERY OF THE INVENTION 
However, in spite of the fact that an environment very close to the 
real space is realized in the virtual space on the computer in the 
25 above-mentioned manner, the number of pointing device that the user 
uses is usually only one. Accordingly, the feeling of the user, even with 
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respect to a work which the user uses both hands to handle in the real 
space, becomes the feeling of performing the work with a single hand 
when the pointing device is used on the GUI and hence, the user receives 
the cumbersome feeling. 

Accordingly, it is an object of the present invention to provide a 
technique on GUI which ensures a manipulation with a feeling close to a 
feeling at the time of performing the manipulation in a real space. 

To achieve the above-mentioned object, following processing is 
performed according to the present invention. That is, a first input 
signal is received from a first pointing device and a display position of a 
first pointer which is displayed on a display device is determined based 
on the first input signal. A second input signal is received from a second 
pointing device and a display position of a second pointer which is 
displayed on the display device is determined based on the second input 
signal. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 is an explanatory view showing the whole constitution of a 
computer system to which a mode for carrying out the present invention 
is applied. 

Fig. 2 is an explanatory view showing an example of a first 
manipulation of the present invention. 

Fig. 3 is an explanatory view showing an example of a second 
manipulation of the present invention. 

Fig. 4 is an explanatory view showing an example of a third 
manipulation of the present invention. 



Figs. 5A to 5B are explanatory views showing an example of a 
fourth manipulation of the present invention. 

Figs. 6A to 6C are explanatory views showing an example of a fifth 
manipulation of the present invention. 

Fig. 7 is an explanatory view showing an example of a sixth 
manipulation of the present invention. 

Fig. 8 is an explanatory view showing an example of a seventh 
manipulation of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
Embodiments to which the present invention is applied are 
explained in conjunction with attached drawings hereinafter. 

Fig. 1 shows a computer system according to this embodiment. 
This computer system includes pointing devices 1 (la/ lb), an electronic 
equipment 2 and a display device 3. The electronic equipment 2 includes 
a connecting portion which connects the electronic equipment 2 with the 
pointing device la, a connecting portion which connects the electronic 
equipment 2 with the pointing device lb, and a connecting portion which 
connects the electronic equipment 2 with the display device 3 respectively. 
These connecting portions are not shown in the drawing. The pointing 
devices la, lb and the display device 3 are connected to corresponding 
connecting portions, respectively. 

In this computer system, it becomes possible to provide a plurality 
of pointing devices 1. In this embodiment, two pointing devices (la, lb) 
are provided. A user of this system is capable of simultaneously 
manipulating respective pointing devices 1 using both hands. Each 



pointing device la, lb includes a mechanism for receiving an input which 
is provided for displacing a pointer displayed on a display screen 31 (see 
Fig. 2) of the display device 3 and a button, although they are not shown 
in the drawing. The pointing device 1 may be, for example, a mouse or a 
5 track ball or the like. 

The electronic equipment 2 includes a CPU (Central Processing 
Unit), a memory and the like not shown in the drawing, and realizes 
following functions by executing a computer program. That is, the 
electronic equipment 2 includes input receiving parts 21 (21a, 21b), a 

10 pointer control part 22, an application 23 and a display control part 24 as 
inner functions thereof. 

The input receiving parts 21 receive inputs from the pointing 
devices 1. In this embodiment, the input receiving parts 21 are 
comprised of two parts (21a, 21b). Respective input receiving parts (21a, 

15 21b) are made to correspond to the pointing devices la, lb. Respective 
input receiving parts 21a, 21b respectively receive inputs from respective 
pointing devices la, lb. To be more specific, the input receiving parts 21 
obtain the displacement information of pointers and the button 
manipulation information from the pointing devices 1 at a fixed period 

20 (for example, 1/60 second). The displacement information is, for 
example, comprised of information which indicates a relative 
displacement quantity on the display screen 31 (see Fig. 2 and Fig. 3) 
which constitutes a two-dimensional plane, while the button 
manipulation information is, for example, comprised of information 

25 which indicates whether the buttons of the pointing devices 1 are in the 
pushed state (ON) or in the non-pushed state (OFF). 
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Here, the number of the input receiving parts 21 is not always as 
same as the number of the pointing devices 1. For example, one input 
receiving part 21 may receive inputs from a plurality of pointing devices 
1. 

The pointer control part 22 obtains the displacement information 
and the button manipulation information which the input receiving parts 
21 receive. The pointer control part 22 holds the positional information 
of pointers 41, 42 (see Fig. 2 and Fig. 3) which are displayed on the 
display screen 31. These positional information are, for example, 
respectively held in the form of XY coordinates on this plane respectively 
using the display screen 31 as a two dimensional X-Y plane. The pointer 
control part 22 calculates the XY coordinates of respective pointers after 
displacement based on the XY coordinates of the pointers 41, 42 and the 
displacement information. The calculated coordinates values after 
displacement are outputted to the display control part 24 as the positional 
information of the pointers 41, 42. Further, this positional information 
and the button manipulation information received from the input 
receiving parts 21 are outputted to the application 23. 

The display control part 24 controls the display device 3 and 
displays an object or the like on the display screen 31. To be more 
specific, the display control part 24 receives the positional information of 
the pointers from the pointer control part 22 and displays the pointers 41, 
42 at the designated positions. Further, upon receiving the drawing 
instruction from the application 23, the display control part 24 displays an 
instructed object or the like on the display screen 31. 

The application 23 performs various processing. Considered here 



is the application 23 which executes a given processing based on the 
instructions from the pointing devices 1 and displays various graphics, 
texts and the like on the display screen 31 is considered. Hereinafter, a 
virtual object or the like which the application 23 displays on the display 
5 screen 31 is called "an object". 

For example, a case in which the application 23 which manipulates 
the object on the display screen 31 as shown in Fig. 2 and Fig. 3 is 
considered. In Fig. % a virtual object 32, a color pallet 33 for selecting 

*t colors and the pointers 41, 42 are displayed on the display screen 31. In 

Q 

OlO Fig. 3, a virtual object 34 and the pointers 41, 42 are displayed on the 

fIJ display screen 31. Here, the pointer 41 and the pointer 42 are 

cil respectively made to correspond to the manipulation of the pointing 

i n 

device la and the manipulation of the pointing device lb. 

The application 23, with respect to the objects such as the virtual 

S3 ■! 

p;! 15 objects 32, 34 and the color pallet 33 and the like on the display screen 31, 
Q holds information indicative of the kinds of objects, information 
indicative of the shapes of objects and the like and information indicative 
of the positions of the objects and the like respectively. Based on these 
information, the application 23 performs the drawing instruction of the 
20 objects to the display control part 24. 

Further, the application 23 receives the positional information of 
the pointers 41, 42 and the button manipulation information from the 
pointer control part 22. Then, based on the positional information of the 
pointers 41, 42, the application 23 grasps the positional relationship with 
25 the objects which have been already displayed. That is, the application 
23 judges whether the pointers 41, 42 are overlapped to the objects which 



6 



I 



have been already displayed or not and determines the position where the 
pointers 41, 42 are overlapped to the objects when the pointers 41, 42 are 
overlapped to the objects. Further, when the pointers 41, 42 are 
overlapped to the existing objects, the application 23 judges the presence 
5 or the absence of the instruction from the pointing devices 1 based on the 
button manipulation information. When the application 23 receives the 
instruction of the pointing devices 1, the application 23 performs a given 
processing. 

For example, in the first manipulation example shown in Fig. 2, the 
10 virtual object 32 and the color pallet 33 are displayed on the display 
screen 31. The user moves the pointers 41, 42 to respective positions 
shown in the drawing by manipulating the pointing devices la, lb. That 
is, the pointer 41 indicates a right upper portion 32a of the virtual object 
32 and the pointer 42 indicates one color region 33a out of the color pallet 
15 3,3. 

Here, when the user pushes the button of the pointing device la, 
the application 23 is informed of the fact that the button of the pointing 
device la is pushed as the button manipulation information. Based on 
this instruction, the application 23 performs a given processing for 

20 selecting the virtual object 32 as a subject object 

Then, when the user pushes the button of the pointing device lb, 
the button manipulation information, which implies the button of the 
pointing device lb is pushed, is communicated to the application 23. 
Accordingly, the color of a region 33a which is indicated by the pointer 42 

25 is specified out of the color pallet 33. As a result, the application 23 
instructs the display control part 24 such that the color of the virtual 
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object 32 which is the selected object is changed to the specified color 
(color which the region 33a indicates). 

Further, in the example shown in Fig. 3, the virtual object 34 is 
displayed on the display screen 31. In response to the manipulation of 
the user, the application 23 performs the processing to rotate the virtual 
object 34 about a designated point which works as the center. 
Hereinafter, a manipulation example to perform this rotating 
manipulation is explained. 

The user manipulates the pointing device la such that the pointer 
41 indicates a point 35 inside the virtual object 34. Further, the user 
manipulates the pointing device lb such that the pointer 42 indicates a 
point outside the virtual object 34. 

Here, when the user pushes the button of the pointing device la, 
the button manipulation information, which implies the button of the 
pointing device la is pushed, is communicated to the application 23. 
Accordingly, the point 35 inside the virtual object 34 is determined as the 
central point of the rotary movement and the application 23 performs a 
given processing to determine the central point of the rotation. 

Then, the user moves the pointing device lb such that the pointing 
device lb draws an arc while pushing the button of the pointing device lb. 
Accordingly, the application 23 receives the button manipulation 
information on the pointing device lb and the positional information of 
the pointer. Here, on the display screen 31, the pointer 42 moves along 
an arc-shaped arrow A. Accordingly, the application 23 instructs the 
display control part 24 such that the virtual object 34 is subjected to the 
rotary movement about the point 35 in response to the movement of the 
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pointer 42. 

With the provision of two pointing devices, the user can perform 
the manipulation of the object on the display screen 31 using both hands 
as described previously. 
5 In the above-mentioned first and second manipulation examples, 

the application 23 does not differentiate the pointer 41 and the pointer 42 
from each other. That is, the role of the pointer 41 and the role of the 
pointer 42 can be exchanged. Accordingly, the function which is 
O performed by using the pointer 41 can be performed by the pointer 42, 
QO while the function which is performed by using the pointer 42 can be 
4* performed by using the pointer 41. 

On the other hand, the application 23 may differentiate the 
° t . pointers 41, 42 from each other. In this case, functions which respective 
:f t pointers 41, 42 can perform are preliminarily allocated to respective 
01 5 pointers 41, 42. For example, one pointer functions as a pointer (the 
pointer 41 in the first and the second manipulation examples) which 
selects an object which becomes a subject or specifies a position which 
becomes the reference, while the other pointer functions as a pointer (the 
pointer 42 in the first and the second manipulation examples) which 
20 performs the instruction of the manipulation using the selected object or 
the specified position or the like as the reference. 

Here, the pointer control part 22 outputs an identifier for 
identifying the pointer to the application 23 along with the positional 
information and the button manipulation information. In the application 
25 23, the pointer from which the application 23 receives the instruction is 
preliminarily set for each processing. The application 23 performs the 
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determination using the identifier of the pointer along with the positional 
information and the button manipulation information. 

Further, the present invention is applicable to various 
manipulation example besides the above-mentioned manipulation 
5 examples. For example, such other manipulation examples are shown in 
Fig. 4 to Fig. 8. 

The third manipulation example shown in Fig. 4 is an example 
which changes the size of an air brush in a painting tool. That is, the 
t pointer 41 selects a location to which the air brush is sprayed. Here, a 
PlO slide bar 51 is manipulated by using the pointer 42 so as to enlarge or 
fli shrink a virtual circle 52 which indicates the range in which the air brush 

as =3 

is sprayed thus adjusting the magnitude of the air brush. 

in 

* ' The fourth manipulation example shown in Figs. 5A and 5B is an 

jK example which performs the deformation processing of the object That 
jy IS is, two points inside a circular object 53 indicated by Fig. 5A are 

C3 designated by the pointer 41 and the pointer 42 and they are pulled in the 

H 

left and the right directions so as to deform the object 53 into a shape 
indicated by Fig. 5B. 

The fifth manipulation example shown in Figs. 6A to 6C is an 

20 example which designates a start point and an end point. That is, in an 
example in Fig. 6 A, at the time of selecting the range of character string, 
the pointer 41 and the pointer 42 respectively designate a start point and 
an end point of the string. In an example in Fig. 6B, at the time of 
drawing a segment, the pointer 41 and the pointer 42 respectively 

25 designate a start point and an end point of the segment. In an example 
in Fig. 6C, at the time of cutting an object, the pointer 41 and the pointer 
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42 designates a start point and an end point of a cutting line. 

The sixth manipulation example shown in Fig. 7 is an example 
which performs other manipulation simultaneous with scrolling. That is, 
while the dragging manipulation is performed using the pointer 41, a 

5 scroll bar is manipulated using the pointer 42 so as to scroll a screen. 

The seventh manipulation example shown in Fig. 8 is an example 
which designates a position in a three dimensional virtual space. That is, 
an X coordinate and a Y coordinate are determined using the pointer 41 
on an X-Y plane and a Z coordinate is determined by manipulating a slide 

10 bar using the pointer 42. The position of a pointer P inside the virtual 
space is determined based on these coordinate values. 

In any one of the above-mentioned manipulation examples, the 
user can efficiently perform the manipulation using both hands. This 
implies that the manipulation environment which approximates the real 

15 space can be obtained even in the virtual space on the computer. 
Accordingly, the operability of the user is enhanced. According to the 
present invention, the manipulation can be performed with the feeling 
which approximates the manipulation in the real space. 
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